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ABSTRACT 


This thesis describes the level of User Information 
Satisfaction with the Composite Health Care System (CHCS). A 
short-form User Information Satisfaction questionnaire, 
developed by Baroudi and Orlikowski (1988), was administered 
to CHCS users at the Naval Hospital, Charleston. 

Dividing the users into three work groups: Physicians, 
Administrative, and Ancillary personnel; statistically 
Significant differences in satisfaction between groups were 
found. Overall, physicians were least satisfied and 
administrative personnel were the most satisfied of the 
groups. 

The respondents as a whole showed noticeable differences 
in satisfaction with factors that comprise User Information 
Satisfaction. The respondents were dissatisfied with the 
factor relating to software contractor’s services. However, 
the users were very satisfied with the factor describing local 


Management Information Department services. 
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I. INTRODUCTION 


A. OVERVIEW 

Organizations of all types are seeking to improve their 
effectiveness and efficiency by using information systems. 
Considerable amounts of time and money are spent to develop 
and implement information systems within an organization. 
Once inplemented, an information system may or may not be 
effective in accomplishing its objective. Thus, an important 
and necessary stage in justifying any particular information 
system is to determine its value of effectiveness to the 
organization. 

The Department of Defense (DoD) has undertaken the 
development and implementation of a large information system 
for use at Medical Treatment Facilities. The system is called 
the Composite Health Care System (CHCS). Due to the costs 
associated with this system, a simple and efficient measure is 
needed to analyze the effectiveness of this information 
system. 

Selecting the appropriate measure of system effectiveness 
is difficult. Much emphasis is placed in many organizations 
on economic analysis of information system. This type of 
analysis provides only an economic measure of effectiveness 


for a system: will the benefit outweigh the cost? If the 


objective of the organization is not purely economic, the 
measure is not complete. 

Many researchers have proposed measuring the level of 
satisfaction perceived by the users of the system as a 
surrogate measure of effectiveness. No matter how technically 
correct an information system, if the users are not satisfied 
with the system, it is not an effective system. Conversely, 
if the users are satisfied, the system is considered effective 


in meeting their needs. 


B. RESEARCH OBJECTIVES 

This study will investigate through empirical research 
whether the Composite Health Care System is effective through 
the use of the user information satisfaction surrogate 
measure. Using a previously developed, documented, and 
validated survey instrument, this study will describe the user 
information satisfaction characteristics at one implementation 
site of the Composite Health Care System: Naval Hospital, 
Charleston. The results of this survey will help pinpoint 
potential problem areas with the system implementation as well 
as document areas without problems. Once completed, the 
results will serve as a baseline gauge to compare satisfaction 
with CHCS at other facilities and to measure changes in 


Satisfaction at Charleston at a later time. 


II. DESCRIPTION OF THE COMPOSITE HEALTH CARE SYSTEM 


A. BACKGROUND 

The Department of Defense (DoD) has pursued the goal of 
providing automated computer support to its hospitals and 
clinics since 1968. In February 1979, the Mission Elements 
Need Statement (MENS), establishing the need for an automated 
computer system was approved. Under the direction of the Tri- 
Service Medical Information System (TRIMIS) program office, 
stand-alone and integrated health care computer systems were 
acquired, implemented and operated to support Pharmacy 
(TRIPHARM) , Laboratory (TRILAB), Radiology (TRIRAD), 
Appointment and Scheduling (TRIPASS), Quality Assurance 
(AQCESS), and Hospital Information System (HIS). The 
knowledge and experience gained through the operation of these 
systems since 1979 was used to refine and validate the system 
requirements for a completely new and fully integrated 


information system: The Composite Health Care System (CHCS). 


B. CHCS PURPOSE 

The primary purpose of CHCS is "to provide health care 
services that support military forces in fulfilling their 
required mission...." (Draft System Decision Paper, 1989). 
Improving the efficiency and effectiveness of military health 


care delivery during peacetime and during mobilization through 


integration of information resources is the goal of CHCS. The 
major focus of CHCS is the integration of shared information 
resources to resolve the deficiencies of communication, 
decision support, and information processing. The Composite 
Health Care System is designed as a fully integrated medical 
information system that provides automated support of 
information requirements for military medical treatment 
facilities. CHCS supports the administrative functions and 
the delivery of health care with information retrieval 


services. 


C. DEPLOYMENT STATUS 

CHCS is undergoing the Operational Test and Evaluation 
(OT&E) phase and is currently deployed to Naval Hospital 
Charleston as one beta-test site. The role of a beta-test 
site is to incrementally replace the TRIMIS systems with 
validated CHCS software. Continued validation of requirements 
and extensive evaluation of the CHCS software is conducted 


prior to the decision to deploy CHCS worldwide. 


D. OPERATIONAL FEATURES 

CHCS supports various functional areas in sharing 
information. Each functional area uses this information for 
its own purpose and communicates results/activities to other 


areas. Functional areas supported by CHCS include: 


1. Patient Administration 
Patient administration does the registration of a 
patient into the system. Once in the computer system, the 
patient’s demographic information is available for access by 
all other modules. The patient administration module also 
performs the functions to admit patients to the hospital, 
transfer patients between wards, and discharge patients from 
the hospital. The creation, update and closing of inpatient 
records is also performed by the patient administration 
module. 
2. Patient Appointment and Scheduling 
The Patient Appointment and Scheduling module provides 
for a centralized appointment service, a decentralized 
appointment service, or a combination of both depending on the 
desire of the hospital. The appointment service creates and 
maintains the appointment schedule for the hospital. A 
patient is given an appointment date and time to see the 
health care provider creating a centralized repository of 
information to produce operational reports of activity. 
3. Nursing 
CHCS supports inpatient nursing in a variety of areas. 
In the area of patient care, nursing uses the system for 
entering and obtaining the status of all physician orders, 
entering patient assessment data, and generating patient care 


plans. Documentation of patient progress is performed in the 


automated nursing notes. Nursing unit management uses CHCS to 
generate shift care plans, patient management reports, drug 
administration time reports, and staffing requirement reports. 
Routine administrative functions are performed by CHCS for 
nursing including: staff credentialling, documentation of 
continuing education and in-service training, and staff 
scheduling capabilities. 
4. lLaboratory 

The laboratory uses the CHCS system for processing of 
orders for laboratory tests with automated reporting of test 
results. Specimens are processed, tracked, and reported 
through the system. All results are placed into the patients 
automated medical record and are immediately available for 
inquiry by health care providers. Automated control of blood 
bank operations including: blood acquisition, inventory and 
utilization are accomplished through CHCS. The laboratory is 
supported with ability to manage inventory control, register 
patients into the tumor registry, conduct a drug testing 
program, and generate a multitude of management reports. 

5. Pharmacy 

The pharmacy uses CHCS to process prescriptions for 
patients that are ordered by health care providers. The 
health care provider enters the prescription information at a 
terminal located in his office. The information is 


electronically transmitted to the pharmacy. The pharmacy can 


immediately prepare the medication for dispensing before the 
patient leaves the physician’s office. CHCS automatically 
performs checks for patient allergies, drug interactions with 
medications the patient is currently taking, drug dosage for 
the age of the patient, and records the prescription into the 
patient’s medical record. In-patient order processing is 
accomplished in much the same _ fashion. On-line drug 
monographs are available for inquiry by medical staff at any 
terminal. Pharmacy management of inventory control and 
reporting are also accomplished. 
6. Radiology 

Orders for radiological procedures are electronically 
transmitted to the radiology department where the order is 
processed. Order tracking and results reporting are performed 
by the system. Radiology results are immediately available 
for inquiry by medical staff. Department management including 
inventory control is also conducted through CHCS. 

7. Clinical Dietetics 

The Clinical Dietetics service uses CHCS to obtain 
patient data needed to access the nutritional needs of the 
patient. This information is used to create patient ee 
plans, select patients of interest, patient menu selections 
and monitor patient nutritional data. CHCS is also used to 
maintain the nutrition clinic schedule and compute clinical 


dietetics workload data. 


E. POTENTIAL BENEFITS 


In general terms, the overall benefits of the CHCS system 


include: 


ie 


reliable, timely, easily accessible clinical 
information 
improved documentation for medicolegal issues 


improved communication 
improved patient satisfaction 


improved management efficiency 


IIIT. USER INFORMATION SATISFACTION AND ITS MEASUREMENT 


A. INTRODUCTION 

Organizations spend considerable amounts of time and money 
to develop and implement information systems. However, since 
financial resources are scarce and limited, not all 
application systems can be developed. Only those applications 
that can be justified are approved and developed. An 
important and necessary stage in justifying any particular 
information system is to determine its value or effectiveness 


to the organization. 


B. METHODS TO ASSESS EFFECTIVENESS 

Several methods have been suggested in the empirical and 
non-empirical literature for assessing the effectiveness of an 
information system. 

System usage (Swanson, 1974; Conrath and Mignen, 1990) 
assumes that a casual relationship exists between 
effectiveness of an information system and the amount of time 
the system is used. An effective information system is used 
more than a less effective system. This method also assumes 
that the use of the system is optional to the user. nee 
however, the use of the system is mandatory, the relationship 


between use and effectiveness fails. 


Others (eg., Hamilton and Chervany, 1981) argue that 
effectiveness is determined by comparing performance to 
ojbectives. To assess the effectiveness of an information 
system, the task objective of the system is first determined. 
Criterion measures are developed to measure the extent to 
which the information system accomplishes the determined 
objectives. Problems are encountered with this method 
because: objectives and measures are often not defined 
adequately; efficiency-oriented and easily employed measures 
are frequently used over effectiveness-oriented and difficult 
measures; individual interpretations of what the ojectives and 
measures are, often conflicting, exist. 

Another method frequently used to determine the value of 
an information system is by economic (cost-benefit) 
evaluation. The benefits of a system, by improved 
organizational effectiveness, are weighed against the cost for 
development and operation of the system (Nolan, 1974). 
Although this appears objective and comprehensive on paper, in 
practice, cost-benefit analysis is highly subjective and 
difficult to conduct (Nolan, 1974). The difficulties in using 
this approach for research argued by Ives et al. (1983) stem 
from the fact that: (1) many costs and benefits are intangible 
and not easily recognizable and/or converted into monetary 
equivalents, (2) unstructured, ad hoc decision making benefits 
are nearly impossible to objectively assess, (3) even when 


these items are determined by an organization, the data are 
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generally unrecorded and not available for research (Ives et 
al., 1983). 

Theoretically, evaluation of the effectiveness of an 
information system is best accomplished by decision analysis, 
based on its degree of use in decision making and the 
resultant productivity benefits attributable to its use 
(Nolan, 1974). An effective information system supports a 
user by retrieving necessary information in the proper format, 
level of detail, and at the proper frequency for use. 
However, the decision analysis approach suffers a significant 
drawback for use in scientific research: it is non- 


quantifiable and cannot be easily replicated. 


C. THE USER INFORMATION SATISFACTION CONSTRUCT 

A surrogate for utility in decision making is the 
satisfaction of users of the information system (Neumann and 
Segev, 1980; Ives et al., 1983; Bailey and Peason, 1983; 
Baroudi and Orlikowski, 1988). 

Webster’s New Collegiate Dictionary defines satisfaction 
as "the fulfillment of a need or want". Organizational 
Psychologist Bernard Bass (1965) generically defines 
Satisfaction to mean "... the extent to which the item is 
rewarding to us relative to how much better we might do 
elsewhere and what aspirations we have." 

The concept of User Information Satisfaction (UIS) can be 


traced back to the work of Cyert and March (1963). The Lr 


idl 


research argues that organizational behavior continually 
imposes upon the manager the need for information. If a 
formal information system exits, the success or failure of 
that information system to meet the needs of the user either 
reinforces or frustrates the user’s sense of satisfaction with 
that system. 

Figure 3.1, an adaptation of the Cyert and March model, 
portrays the reinforcement/reduction of satisfaction process 
in decision making using the information system. When a user 
perceives that the needed information should be readily 
available in the formal information system, he will use the 
information system in the normal manner to retrieve the 
required information. If the information is readily 
available, satisfaction with the system will be reinforced. 
But, if the information is not readily available, he must 
adapt and conduct a search beyond the information system by ad 
hoc methods or by some other less familiar method. 

An "expanded search" through the information system is a 
cause of frustration to the user because of the time and 
effort used to obtain the necessary information. 
Additionally, the user is frustrated because he perceived the 
information to be readily available when it was not. From 
users’ standpoint, information systems may actually block the 
user from carrying out the act of decision making (Nolan, 


EIS): 
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is there a decision 
to be made? 
Yes | 


is the information 
readily available Expand search 


in the Inforamtion beyond present 


a 


system? information 


Reinforcement of Reduction of 
satisfaction with satisfaction with 
the information the information 
system system 





Figure 3.1 
UIS reinforcement /reduction 
in decision making 


The information system is continually being evaluated by 
the user from repetitive use. If over a period of time of 
use, the user perceives that the information system cannot 
retrieve the information without the frequent use of expanded 
searches, frustration will occur. Consequently, this may lead 
to an expression of user dissatisfaction with the information 
system. The project is a failure if the end product does not 
satisfy the manager whom it is to serve (Powers and Dickson, 


1973). If, however, the user is able to reliably obtain the 
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needed information with normal operation of the information 
system, satisfaction with the system will be reinforced and 
the information system is successful. 

Of note are two empirical studies, (Aldag and Power, 1986; 
Gallupe and Desanctis, 1988), undertaken to find a 
relationship between Satisfaction and actual system 
performance using objective third party judges. The results 
of these studies, however, are conflicting but do not support 
the assumption that increased satisfaction indicates actual 


system performance increases. 


D. THE MEASUREMENT OF USER INFORMATION SATISFACTION 
Potentially measurable and generally acceptable, several 
studies have employed user information satisfaction (UIS) as 
a dependent variable to indicate system effectiveness and 
acceptance (Ives and Olson, 1979; Igbaria and Nachman, 1990). 
Neumann and Segev (1980) show a correlation between user’s 
reaction to satisfaction factors and their perception of the 
organizations’s performance. Swanson (1974) empirically found 
high correlation between the user’s appreciation of the system 
and his/her usage of the outputs. Powers and Dickson (1973) 
concluded that user satisfaction is the most critical factor 
for system success. Although there is no standard measure of 
satisfaction in these studies, it is argued that user 
information satisfaction is an indicator of system usage and 


success. 
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The level of user information satisfaction in the studies 
is derived from a myriad of factors when users were asked to 
evaluate their computer services relative to a sense of 
satisfaction (Bailey and Pearson, 1983). Factors measured in 
the various studies include: accuracy, content, frequency, 
timeliness, reliability, assistance, adequacy, accommodation, 
communication, access, appreciation and flexibility. Each of 
these studies use a measure that is unique to that particular 
Study (Ives et al., 1983), but taken as a whole provide 
insight into a description of UIS. 

Validation of the UIS measures in these studies is 
limited, especially in handling threats to internal validity 
(Haga and Zviran, 1990). However, a tool for measuring and 
analyzing UIS was developed by Bailey and Pearson (1983), 
based on the previous studies, that provides the basis of an 
instrument, later refined (Ives et al., 1983), and validated 
(Baroudi and Orlikowski, 1988) for use. Although the method 
employed to measure user satisfaction by this tool is a 
pre-experimental design lacking pretest-postest or control 
group (Campbell and Stanley, 1966), Conrath and Mignen (1990) 
claim it represents a key contribution in the development of 
a standard instrument to measure user satisfaction. The 
development and refinement of this instrument will be 
described in more detail. 

Bailey and Pearson (1983) described a model proposed by 


Lawler and Wanous (1972) for measuring user satisfaction: 


i 


j=1 
where 
R,, = The reaction to factor 7 by the individu 
Wi, = The importance of factor j to individual i 


Using this model, the satisfaction of the user is measured 
as the weighted sum of the user’s positive and negative 
reactions to a set of factors about the information system 
(Bailey and Peason, 1983). The user’s perception of a "good" 
information system would be a system that the user is very 
satisfied with the factors considered most important in the 
system. 

To implement the model, the set of factors representing 
the domain of user satisfaction was established via a review 
of 22 studies dealing with computer/user interface (Bailey and 
Pearson, 1983). The initial 36 factors generated from the 
literature review were expanded to 39 factors after further 
review by middle managers. They concluded that the 39 factors 
included represent the domain of user satisfaction at a=0.01. 

Bailey and Pearson (1983) used four bipolar adjective 
pairs on a seven point Likert-type scale to measure the user’s 
perception of a factor. Along with the four adjective pairs 
was included the user’s perception of importance of the 
factor. The seven intervals from negative to positive were 


denoted by adverbial qualifiers. Figure 3.2 represents their 
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use of the semantic differential technique for measuring the 


user’s response to "Format of Output" factor. 


Format of Output: The material design of the layout and 
display of the output contents. 


geod a: Se Ge ee: Se: aesceee : bad 
Solem i Cr 1) >  : complex 
Reddawlen: i: Se : 9: “ws : unreadable 
useful : ee: : | a : : useless 
To me, this factor is ea = 
important : : : : : : : : unimportant 


Figure 3.2 Illustration of questionnaire form 


The Bailey and Pearson user information satisfaction 
questionnaire was determined to be ae reliable, valid 
instrument (Bailey and Pearson, 1983). It represented an 
important first step toward the development of a valid and 
useful UIS measure (Ives et al., 1983). 

Deese (1979) used the Pearson questionnaire at the Federal 
Computer Performance Evaluation and Simulation Center. He 
stated, "The results identified problems that would not 
otherwise have been discovered" and claimed that the user 
satisfaction questionnaire was a very useful and worthwhile 
tool (Deese, 1979). 

Ives, Olson and Baroudi (1983) chose to undertake an 
in-depth assessment of the Pearson (1983) questionnaire. 
Replicating the initial Bailey and Pearson study, they were 


able to reinforce the validity and reliability of the 
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instrument. They presented several approaches to improve the 
quality of the original Pearson instrument. 

The goal of the suggested improvements was to establish a 
standardized "short form" instrument. Since Pearson found the 
importance scale provided no additional information, it was 
eliminated from the instrument. Scales which showed 
undesirable psychometric qualities were also eliminated. The 
number of items scaled within a question was reduced from four 
to two in order to reduce the time to complete the 
questionnaire. All of the scales in the original Pearson 
instrument were scored positively to the left and negatively 
to the right end of the scale. To reduce the undesired 
effects created by a person simply marking down a column of 
responses, some of the scales were reversed scored, thereby 
increasing the reliability of the measure. The resulting 
short form was determined to substantially measure the concept 
of UIS originally proposed in the Pearson full instrument 
(Ives et al., 1983). 

The short form measure (Ives et al., 1983) was further 
subjected to testing by Baroudi and Orlikowski (1988) to 
examine its psychometric properties. The final short form 
questionnaire consisted of 13 questions with two items 
evaluated per question (Appendix A). Each item is scored on 
a seven point Likert-type scale from -3 (dissatisfied) to +3 


(satisfied) with zero indicating a neutral response. 


18 


The total individual user information satisfaction score 
is calculated by averaging the responses of the two items for 


each question and summing the scores of the 13 questions. 





13 
R,+R 
sy a 2 
1 


where 

S, = total individual user satisfaction 
R, = response to the first item scale 
R, = response to the second item scale 


The range of total satisfaction can be from -39 to +39. 

Three factors; electronic data processing (EDP) staff and 
services (Factor A), information product (Factor B), and 
knowledge and involvement (Factor C); were found to comprise 
user satisfaction using factor analysis by Ives, Olson and 
Baroudi (1983). These subtotals are calculated as the average 
of the responses to questions loading into a particular 
factor. Questions 1, 2, 6, 11, and 12 load heavily into the 
electronic data processing (EDP) staff and services factor. 
Questions 7, 8, 9, 10 and 13 load heavily into the information 
product factor. Questions 2, 3, 4, and 5 load heavily into 
the knowledge and involvement factor. The factor subtotals 
will range from -3 to +3 in value. Averages are used for 
meaningful comparison between the three factor scores. 

Data were gathered for the study from 358 employees, 


mostly clerical and support personnel, of 26 New York area 


ee, 


organizations. Construct validity was determined first by 
examining the relationship between each scale and the total 
UIS score. Factor analysis using varimax rotation converged 
to a three factor solution accounting for 68% of the variation 
in five iterations. This provides strong evidence of 
construct validity (Baroudi and Orlikowski, 1988). 

Convergent validity was proven by comparing an interview 
measure of satisfaction with the instrument measure in two 
groups. One group of users were generally pleased and 
satisfied with the information system and the other group of 
users were generally dissatisfied on interview. 
Administration of the instrument indicated a statistically 
Significant difference by t-test in the groups at p < .001 
(Baroudi and Orlikowski, 1988). 

The total satisfaction and subtotal scores reported a 
reliability level above the .80 required for research. This 
demonstrates the instrument is internally consistent and 
reasonably free from measurement error. 

The work by Baroudi and Orlikowski (1988) and resulting 
short-form UIS instrument provides a reliable and valid 
measure of user information satisfaction. It furnishes a 
meaningful, standardized measure of the overall satisfaction 
with an information system as well as specific information 
about satisfaction within the subfactors of electronic data 
processing (EDP) staff and services, information product, and 


user knowledge and involvement. 
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i Oe RESEARCH METHODOLOGY 


A. INTRODUCTION 

The purpose of this research is to identify the 
characteristics of user information satisfaction (UIS) for 
users of the Composite Health Care System (CHCS). The study 
is based on statistical analysis of empirical data collected 
for CHCS users at the Naval Hospital, Charleston, SC. The 
following sections will discuss the survey instrument, data 
collection methodology and methods of statistical analysis 


employed in the study. 


B. THE SURVEY INSTRUMENT 

To avoid the pitfalls associated with developing a 
comprehensive survey questionnaire, the previously developed, 
psychometrically evaluated and validated short-form 
questionnaire of Baroudi and Orlikowski (1988) was used 
without alteration. A copy of the complete survey 
questionnaire is included in Appendix A. The questionnaire is 
composed of two sections: a demographical section and a user 
information satisfaction section. 

1. Demographic Section 

The first section of the questionnaire requested 

general information. Questions 1-5, the respondents were 


asked to give their hospital department, job description, 
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highest level of education, age, and gender respectively. To 
determine the user’s experience level with the system, the 
sixth question asked for the length of time, in months, the 
respondent had used CHCS. Further determination of the user’s 
experience was sought in questions seven and eight. Question 
seven asked if the user had used other computer systems 
previously. If the respondent answered "yes" to question 
seven, he or she was asked if the previous system was a health 
care information system. 
2. User Information Satisfaction Section 

The second part of the survey directly addressed the 
issues of user information satisfaction. The instrument was 
designed to elicit the user’s perception of how well the CHCS 
system functioned in their area. It consisted of thirteen 
questions; each question having two bipolar adjective Likert- 
type item scales for responses. Each item can take on one of 
seven values from -3 (extremely dissatisfied) to +3 (extremely 
satisfied) with zero indicating a neutral response. 

The questions asked can be categorized into one of 
three factors of user satisfaction. 

Management Information Services (MIS) Staff and 
Services: This factor is the respondent’s perception of the 
attitude and responsiveness of the MIS staff and the 
relationship with the MIS staff. Due to the fact that the MIS 


personnel at the Naval Hospital, Charleston provide only local 
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Support and act as liaison to the contractor that supplies the 
software product, this factor is further broken down to 
reflect services provided by the local MIS department and the 
contractor. 

Information Product: This factor is the respondent’s 
perception of the quality of output delivered by CHCS. 

Knowledge and Involvement: This factor is the 
respondent’s perception of the quality of training provided, 
their understanding of the system, and their participation in 


its development. 


C. SAMPLE AND DATA COLLECTION 

Naval Hospital, Charleston is one of thirteen beta-test 
sites for CHCS software. It was selected as the site to 
conduct the survey because all functional modules were to be 
operational at the time of the survey. Since the in-patient 
modules, servicing nursing and clinical dietetics, were not 
implemented at the time of data collection, the sample 
population then consisted of personnel from out-patient and 
administration areas. 

A local point of contact acted as a distribution and 
collection agent for the surveys. The questionnaires, each 
accompanied with a cover letter (Appendix B), were mailed to 
the point of contact at the site. The point of contact 
distributed 180 questionnaires and return envelopes to all 


department personnel within the hospital operating the CHCS 


Ze 


modules. The surveys were completed and returned by the 
respondents to the point of contact in sealed envelopes and 
then the questionnaires were forwarded to the researcher for 
analysis. 

110 questionnaires were returned, of which 101 surveys had 
complete information. This represents a final return rate of 


36%. 


| D STATISTICAL ANALYSIS METHODS 
Data from completed surveys were coded and entered into a 
spreadsheet program. The spreadsheet program was used to 
perform the statistical analysis. The specific procedures 
used in the analysis will be presented next. 
1. Respondent’s Satisfaction Index 
Each respondent’s total satisfaction index was 


calculated for the 13 questions using the following formula: 


iL: 
ous Raiythyiy 
j= * 
where: 
Si = Satisfaction Index for respondent i 
Rij = Response to first item scale of question j for 


repondent i 
Response to second item scale of question j for 
respondent 1 


Rois 
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2. Total Satisfaction Index 

The overall or total satisfaction index for the survey 
group is calculated by averaging the respondent’s satisfaction 
index to find the mean. 

3. Question Averages 

The mean response to each of the thirteen questions is 

calculated for comparison of individual questions. 
4. Factor Subtotals 

The thirteen questions can be grouped into three 
factor subtotals as was defined by Ives, Olson, and Baroudi 
(1983): MIS staff and services (Factor A); Information product 
(Factor B); and Knowledge and involvement (Factor C). The 
mean of the individual questions averages is calculated to 
find the factor subtotals. Questions 1, 2, 6, 11, and 12 are 
used for the MIS staff and services subtotal. Questions 7, 8, 
9, 10, and 13 are used for the information product subtotal. 
Questions 2, 3, 4, and 5 are used for the knowledge and 
involvement subtotal. 

The MIS staff and services sub-factor is further 
subdivided to provide insight into the distinction between the 
local Management Information Department (MID) department 
functions and the functions of the software contractor. 
Again, the mean of the individual question averages is 
calculated to subdivide the MIS staff and services factor. 


Questions 1, 6, and 11 directly address the attitude, services 
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and relationship with the local MID are used for the local MID 
factor. Questions 2 and 12 address issues associated with 
software developement, the responsibility of the civilian 
contractor, and are used for the contractor factor. 
5. Comparison of Groups 

The respondents were categorized by the type of use of 
the CHCS system. Three work groups were identified: (1) 
Physicians; (2) Ancillary; and (3) Administration. The 
individual question averages, total satisfaction indexes, and 
sub-factor totals were compared for one group verses the 
combination of the other two groups using a t-test with the 
Significance level of alpha = 0.05. The t-test is used to 
determine if there is a significant difference between the 


arithmetic mean value of two groups. 
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V. DESCRIPTIVE FINDINGS 


A. DEMOGRAPHIC FINDINGS 

Of the 101 respondents, 57 were male and 44 were female. 
The age of the respondents ranged from 19 to 56 with a mean of 
32 years old. 

lL. Work Center Distribution 

The hospital department work centers reported were 

out-patient areas including: administration, clinics, pharmacy 
and laboratory departments. The CHCS modules for in-patient 
areas were not operational at the time of the _ survey. 
Personnel from in-patient areas were not included in the 


study. Figure 5.1 reflects the distribution by work center. 
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Figure 5.1 
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Job Description Distribution 
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Figure 5.2 
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3. Work Group Distribution 
For the purposes of this investigation, the study 
population was segregated according to the assumed primary use 
of the computer system. The categories provide a clear 
separation of work groups from an organizational and cultural 


standpoint. These groups are used extensively throughout the 


study. 

1. Administrative - use of the system for indirect 
medical purposes (patient registration, 
appointment scheduling and administrative 
reporting) 

2. Ancillary - use of the system for direct medical 


Support (prescription filling, laboratory specimen 
processing) 
3. Physician - use of the system by physicians 
(accessing patient data, transmission of orders) 
Figure 5.3 represents the population of respondents according 


to their main use of the system. 
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4. Education Distribution 
The level of education reported by the respondents 
represents a well educated user population. Almost half of 
the users have a college degree and almost ninety percent have 
had some college education. Figure 5.4 represents the 


breakdown by education. 
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Figure 5.4 


B. COMPUTER SYSTEM USE 

The length of time the respondents reported as having used 
the CHCS computer system ranged from one month to 18 months 
with a mean of 8.7 months. Of the 101 respondents, 83 had 
used a computer system previously. Only 24 of the 83 users 
had used a health care information system before the CHCS 
implementation. Although many of the respondents had 
experience with computers, this was the first exposure to a 


health care information system for 76 percent of the users. 


C. SATISFACTION FINDINGS 
The survey results for user satisfaction are compared for 


the three work groups in three areas: 1) overall satisfaction, 


Sil 


2) responses to individual questions, 3) three group factors: 
MIS staff and services, information product, and knowledge and 
involvement. The actual values reported in the survey 
responses are not significant in themselves, rather the 
relative scores among individual questions, sub-factors and 
work groups are useful for comparison. Since ancillary 
composes 46% of the survey group, the survey averages will 
tend to more closely reflect the responses of the ancillary 
group than the other groups. Therefore, average scores on 
individual questions are not fully representative and are 
included only for rough comparisons. 
1. Overall Satisfaction 

Overall satisfaction is measured by the mean of the 
sum of the responses to the survey questions. The mean survey 
score index was 11.26 on a scale ranging from -39 to +39. 
Figure 5.5 presents a comparison of the overall satisfaction 
index the three groups and the overall mean satisfaction. The 
survey shows that the administrative group displays the 
highest overall satisfaction (14.96), followed by ancillary 
(S265); with physicians recording the least overall 
satisfaction (8.07). Using a t-test, there is no significant 
difference between any one work group and the remaining groups 


at alpha = 0.05. 
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Figure 5.5 


2. Individual Questions 
Each of the 13 questions on the survey has two 
responses from the user. The response scales were reversed 
scored on many of the questions to neutralize the effect of an 
individual marking boxes straight down the questionnaire. 
Figure 5.6 shows the average level of satisfaction for 


individual questions by work group. 
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Individual Question Response 
By Work Group 
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Figure 5.6 


It is readily apparent that some questions exhibit a 
relatively high or low level of satisfaction when compared to 
the other questions. Three questions (question 1, 6 and 11) 
stand out from the others for the high level of satisfaction 
expressed by the users. Each of these questions deal with 
issues concerning the Management Information Services (MIS) 
staff. Two questions stand out as exhibiting an extremely low 
(negative) satisfaction index. Questions 2 and 12 are 
concerned with software development time. A complete 
decomposition of each question by work group will follow. 

Figure 5.7 displays the responses to individual 
questions by physicians compared to the remainder of the 
respondents; Figure 5.8 administrative compared to the 


remainder; Figure 5.9 ancillary compared to the remainder. 
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Figure 5.7 


Administration vs Others 
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Figure 5.8 
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Figure 5.9 


a. Question 1 - Relationship with the Management 
Information Department (MID) Staff 
This question measures the level of constructive 
interaction between the user and the Management Information 
Department. As noted earlier, the level of satisfaction 
towards the MID staff is relatively high. Although not 
statistically significant at alpha = 0.05, the administrative 
group showed the highest level of satisfaction of the three 
groups. Slightly lower was the ancillary group followed by 
the physicians. 
b. Question 2 - Processing of Requests for Changes to 
Existing Systems 
The requested responses were concerned with the 


length of time required and timeliness of changes to the CHCS 


3.6 


system. The overwhelmingly negative level of satisfaction; 
the lowest scoring question; indicates a perceived problem by 
the users. Ancillary users were the least satisfied, 
physicians slightly higher and administrative users, albeit 
still negative, the highest group. A t-test indicated no 
Significant differences at alpha = 0.05. 
c. Question 3 - Degree of Training Provided to Users 
This question asked if the users were completely 
trained to use the system. The level of satisfaction of the 
administrative group was statistically higher at alpha = 0.05 
than the other groups combined. Correspondingly, the 
ancillary group was significantly less satisfied with the 
training. The physicians had the lowest mean score but did 
not prove to be statistically different. 
d. Question 4 - User’s Understanding of the System 
The level at which a users understands a system 
naturally flows from training and experience. As expected, 
the response to this question closely parallels the users 
perception of the degree of training provided. In this case, 
physicians were significantly less satisfied at alpha = 0.05. 
Of particular note is the significant improvement in 
satisfaction in the ancillary group from their satisfaction of 


training. 


S77 


e. Question 5 - User’s Feeling of Participation 
The user’s perception of participation is 
beneficial for the success of a computer system. Lack of 
participation may lead to dissatisfaction with the system. In 
response to this question, the administrative group had a 
statistically significant higher feeling of participation than 
the other groups. Physicians perceived the least amount of 
participation. Ancillary personnel responded more positively 
than physicians but less positively than administrative. 
f£. Question 6 — Attitude of the Management Information 
Department (MID) Staff 
The willingness and commitment of MID staff to lend 
assistance for the benefit of the users is measured by their 
perceived attitude towards the users. The overall response to 
this question (the highest rated question) indicates that MID 
displays a highly cooperative and positive attitude. All 
three groups responded favorably. However, administration 
reported a statistically significant (alpha = 0.05) higher 
response than the other groups. Although the physicians 
scored this question the highest of all 13 questions, their 
response was statistically lower than the other groups. 
g. Question 7 —- Reliability of Output Information 
Users must be able to accept the output information 
provided by the computer system as reliable. This is 


extremely important for medical information systems. The 
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responses were positive with no significant differences among 
the groups. Of note is the fact that physicians responded 
more favorably than the other groups and ancillary the least 
favorably. 
h. Question 8 —- Relevancy of Output Information (to 
intended function) 

This question measures the degree of congruence 
between what the user wants or requires and what is provided 
by the information products and services. All groups 
responded positively without statistically significant 
differences. The administrative group was most satisfied with 
the relevancy of output of the three groups. 

i. Question 9 — Accuracy of Output Information 

All three groups were satisfied that the system was 
providing correct information. Of the three groups, the 
physicians were the most satisfied with the accuracy of output 
information. Ancillary and administrative groups were about 
equally satisfied. The differences in responses were not 
statistically significant. 

j- Question 10 - Precision of Output Information 

Precision of output information is the variability 
of output information from that which it purports to measure. 
The response was favorable, but was the lowest of the 


questions answered positively. The ancillary group responded 
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statistically lower than the other groups on this question 
while the physicians recorded the highest score. 
k. Question 11 - Communication with the Management 
Information Department (MID) Staff 
Communication is the manner and methods of 
information exchange between the users and the MID staff. As 
noted previously, all questions concerning MID staff received 
high satisfaction levels from all the groups. None of the 
groups’ responses were statistically different by t-test at 
alpha = 0.05. 
1. Question 12 - Time Required for New Systems 
Developement 
New systems development time is the time between 
user-initiated request for service or action and a reply to 
that request. Much like the processing of requests for 
changes to existing systems, the groups were either neutral or 
highly dissatisfied. The physicians were statistically less 
satisfied than the other groups; reporting the lowest index of 
any question. 
m. Question 13 -—- Completeness of Output 
All three groups were adequately satisfied with the 
comprehensiveness of the content of information product. The 
level of satisfaction of each group was similar with the 


administrative group the most satisfied. 
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3. Grouped Factors 

The questions are grouped into three factors that 
comprise user satisfaction described by Ives, Olsen, and 
Baroudi (1983). These factors: MID staff and services, 
information product, and knowledge and involvement, are 
calculated through averages of the component questions. MID 
staff and services (Factor A) is sub-divided into two separate 
components: local MID staff functions (MID) and contractor 
functions (Contractor). Factor A (MID) is derived from 
questions 1, 6,and 11; Factor A (Contractor) from questions 2 
and 12; information product (Factor B) from questions 7, 8, 9, 
10 and 13; and knowledge and involvement (Factor C) from 
questions 2, 3, 4, and 5. 

None of the work groups’ satisfaction level to any of 
the factors were statistically different at significance level 
alpha = 0.05. However, general observations can be made. 
Figure 5.10 represents the work groups’ level of satisfaction 


with relation to sub-total factors. 
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Figure 5.10 


Figure 5.11 represents factor satisfaction of 


physicians compared to other respondents. 


Grouped Factors 
Physicians vs Others 


5 
z 
5 
g 
Az) 
3 
Y 


Contractor 


BES Phystolan [Others 





Figure 5.11 
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Figure 5.12 represents factor satisfaction 


administrative personnel compared to other respondents. 
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Figure 5.12 


Figure 5.13 represents factor satisfaction 


ancillary personnel compared to other respondents. 
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Figure 5.13 


a. Factor A (Local MID Staff and Services) 

This factor represents the attitude, services and 
relationship with the local MID personnel. As seen in Figure 
5.10, the administrative personnel displayed the highest level 
of satisfaction. The lowest level of satisfaction, although 
still extremely positive, was displayed by the physicians. 

b. Factor A (Contractor Services) 

This factor represents the responsiveness and 
services of software programmers. This factor received the 
lowest level of satisfaction of any factor. All work groups 
rated this factor negatively, with ancillary personnel 


recording the lowest level of satisfaction of any group. 
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c. Factor B - Information Product 
This factor represents the users perception of the 

quality of output from the system. Although the differences 
among the groups were not statistically significant, the 
administrative group scored the highest level of satisfaction. 
Ancillary members presented the lowest level of satisfaction. 

d. Factor C - Knowledge and Involvement 

This factor represents the respondents’ self- 
reported assessment of the quality of training provided, 
their understanding of the system, and their participation in 
its development. There was a statistically significant 
difference between the level of satisfaction of the 
administrative group and the other respondents. 
Administrative personnel were highly satisfied with their 
knowledge and involvement of the system. 
4. Tame of System Use 

As users become familiar with the functions of a 
system over time, it is expected that the more experienced 
users would exhibit a greater level of satisfaction. 

Separating the respondents into subgroups of users 
with experience less than six months, six months to 11 months, 
and greater than 11 months provides subgroups of approximately 
equal size. The work groups exhibit roughly the same 
PeopoOrtions. Figure 5.14 represents the breakdown of 


respondents time of system use. 
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Figure 5.14 


All the respondents, broken down by time of system 
use, demonstrate that user satisfaction did not appreciably 
change with the increase in experience of users. Figure 5.15 
shows the level of satisfaction with CHCS verses time of 


system use. 
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Figure 5.15 


None of the work groups; administrative, physicians, 
or ancillary; exhibited any high correlation between time of 
system use and level of satisfaction of any factor by 
regression analysis. However, it is noteworthy to report the 
trends in the level of satisfaction for the work groups over 
time. 

The administrative work group respondents with less 
than six months system use report a satisfaction index of 
8.71. As the users gain experience with the system of six 
months, the level of satisfaction increases to 17.42. 
Satisfaction remains consistent, at 17.29, for users with 
greater than 11 months experience. Figure 5.16 shows the 
relationship of the satisfaction of administrative users to 


time of system use. 
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Figure 5.16 
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Figure 5.17 


Unlike either the administrative or physician work 
groups, ancillary respondents with the least experience in 
system use demonstrated the greatest satisfaction 
(satisfaction index 16.90). The level of satisfaction 
declined to 11.44 for users with six to 11 months experience. 
Satisfaction further declined to 1.56 for users with greater 
than 11 months experience. Figure 5.18 shows the level of 


satisfaction verses time for ancillary users. 
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Figure 5.18 
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VI. ANALYSIS OF FINDINGS 


A. INTRODUCTION 

The purpose of this research is to describe and document 
the user information satisfaction characteristics of the users 
of the Composite Health Care System at the Naval Hospital, 
Charleston. The study results are not to be seen as a 
definitive evaluation of CHCS, but provide a baseline 
reference point for identifying system strengths or possible 
areas of conflict and dissatisfaction. Areas identified as 
dissatisfying should be examined further through interviews 
and by examining the development and operating procedures of 
the work group to determine the possible causes and course of 
action. 

The value of the satisfaction scores (positive or 
negative) is an important finding in itself, however, the 
relative comparison of scores across different groups of users 
is useful for the purposes of this study. 

The analysis of the survey findings will primarily focus 
on the differences and similarities among the three work 
groups: administrative, ancillary, and physicians. The first 
section will look at the overall user information satisfaction 
wath CHCS. The second section will discuss’ significant 


differences in individual question responses reported between 


Sk 


work groups. The third section will compare the three work 
groups with respect to the grouped factors: local MIS staff 
and services, contractor services, knowledge and involvement, 
and information product. The last section will look at user 
satisfaction for the three work groups from the perspective of 


time of system use. 


B. OVERALL SATISFACTION 
1. Combined Groups 
The overall satisfaction index is a_ general 
description of the user’s perception of CHCS as a whole. A 
negative result index would indicate dissatisfaction while 
positive results index some degree of satisfaction with the 
system. The survey average satisfaction index reported is 
11.26. Since the overall satisfaction index ranges from -39 
to 39 in scale, the respondents average overall satisfaction 
is characterized as slightly satisfied using the adjective 
qualifiers of Baroudi and Orlikowski (1988). 
2. Comparison of Work Groups 
The three work groups used in this study were chosen 
to differentiate the users by the way they primarily use the 
computer system. There were no significant differences found 
in overall satisfaction between any one work group and the 
remaining groups. Although all groups are considered slightly 
satisfied with CHCS, there are meaningful differences between 


the work groups’ overall satisfaction. 
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Lowest of the work groups in overall satisfaction was 
the physician group (8.07). Physicians use CHCS almost 
exclusively for the conduct of direct patient care. The 
functionality of the system for physicians is the most diverse 
of the three groups. Requirements of computer input are 
different for medical record queries, prescribing medications, 
laboratory tests, radiology requests, and so on. The 
physician needs to know the procedures for each type of 
functionality to use the system effectively. 

Each physician has a terminal located in the office. 
When using the system for out-patient purposes, the patient 
frequently is present in the office at the time of use. In 
addition, the medical profession have not traditionally used 
automated means to document patient treatments or write 
prescriptions in the past. The computer system can be 
considered, by them, an intrusion into their already busy 
schedule and has caused the physicians to alter their work 
patterns to accommodate the system. 

The significant change of work habits, extensive 
functionality requirements, time constraints imposed on the 
physicians, and the presence of patients in the office while 
operating the system combine to make this finding not 
surprising. 

Ancillary personnel, the largest of the work groups 
(46%), demonstrated a slightly higher overall satisfaction 


score (9.65). Most of the users in this work group are 
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technicians using the system to process prescriptions, 
laboratory specimens, or radiology requests. Ancillary areas 
in the hospital have had automated processing capabilities, in 
some form or another, for many years and are very accustomed 
to general computer use. Due to the large number of items 
processed each day in these areas, the users are under a tight 
time constraint to process items through the computer quickly. 

The administrative group scored the highest overall 
Satisfaction scores. Administrative personnel use the system 
for purposes other than for the direct conduct of patient 
care. With a few exceptions, the work performed on the system 
can be accomplished without the pressure of a limited and 
specific time constraint. The computer system can therefore 
be used at a comparatively slower pace than the other work 
groups. Once again with a few exceptions, most of the work 
performed by the administrative personnel previous to CHCS was 
performed manually. Automating the compilation and generation 
of reports can significantly reduce the time spent on these 


activitics-: 


C. SIGNIFICANT INDIVIDUAL QUESTION DIFFERENCES 

Although there are demonstrated differences in 
satisfaction among the various groups in every question, only 
the questions with significant differences among the work 


groups will be discussed in this section. The individual 
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questions will be combined into the grouped factors 
representing satisfaction in the next section. 
1. Degree of Training Provided to Users 

The results of the survey show the physicians were the 
least satisfied with the training provided, followed closely 
by the ancillary personnel. There is a significant increase 
in the satisfaction with user training expressed by the 
administrative personnel. 

Effective user training is extremely important to the 
success of any computer system. The person’s satisfaction 
with the initial user training received sets the stage for his 
or her satisfaction while operating the system. Often, user 
training is intensive only during the time a system is being 
implemented. After implementation new users are frequently 
required to learn the functionality of a computer system with 
On-the-job-training (OJT). Formalized training programs are 
difficult to conduct on a regular basis due to the constant 
change-over of personnel experienced at medical treatment 
facilities and the specialized requirements of the different 
work groups. Satisfaction with the degree of training 
provided is closely associated with the user’s understanding 
of the system and their feeling of participation. 

2. User’s Understanding of the System 
Users generally attain a high level of understanding 


of a computer system from either effective training or 
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experience or both. As anticipated, the work groups that 
reported a low level of satisfaction in user training 
(Physicians in particular) demonstrated a significantly lower 
level of satisfaction with their understanding of the system. 
The diverse functionality of the module used by physicians may 
also diminish their perception of the level of understanding. 
3. User’s Feeling of Participation 

The user’s perception of the shared involvement and 
commitment to the operation of the computer system is 
beneficial to the success of a computer system. Satisfaction 
with the level of participation naturally follows from the 
satisfaction of user training and the relationship with the 
MID staff. It is not surprising to observe the administrative 
work group scoring significantly higher than the other groups 


on this question. 


D. GROUPED FACTORS 

Baroudi and Orlikowski (1988) argue that the individual 
questions can be grouped into three factors, MIS staff and 
services, information product, and knowledge and involvement, 
that comprise user satisfaction. As discussed in Chapter V, 
the functions normally provided by a MIS of the companies 
studied in their research are provided by two separate 
entities at Naval Hospital, Charleston. Local support 
services are provided by the Command’s Management Information 


Department, whereas software support is furnished by a 
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Civilian software contractor. It is logical to divide the 
single factor, MIS staff and services found in the original 
study, to make two separate factors: local MID staff and 
services and Contractor’s services. 

The three work groups generally scored similarly in each 
of the four factors. The only exception being the 
statistically significant high level of satisfaction of the 
administrative group for the knowledge and involvement factor. 

The most notable finding when four factors are examined is 
the significant negative level of satisfaction of all the work 
groups with the services provided by the software contractor. 
The services specifically asked in the questionnaire dealt 
with satisfaction of the time for system changes and new 
system development. The negative satisfaction finding for 
this factor is, however, expected. Bailey and Pearson (1983) 
list these two factors as the two most frequent for causing 
dissatisfaction among users; a finding supported by other 
published reports. 

At the other end of the spectrum, the user’s perception of 
the local MID staff and services is considered quite 
satisfactory. There does not appear to be any communication 
or attitudinal barriers to prevent an interchange of ideas 
between the respondents and the local MID staff. 

Lower in satisfaction is the user’s perception of the 
information product factor. The information product measures 


the quality of the information output provided to the user. 


oy 


The three work groups combined rated this factor as slightly 
Satisfactory. The users perceive the system positively in 
regards to the information’s accuracy, variance, reliability, 
and relevancy to the intended function. 

The physician and ancillary work groups displayed a 
positive satisfaction index for the knowledge and involvement 
factor. However, the satisfaction level demonstrated, albeit 
positive, was very close to neutrality. Administrative 
personnel indicated a satisfaction level significantly higher 
than the other two groups; high enough to be considered 
slightly satisfied with the factor. The combination of 
satisfaction with user training, system understanding, and 
participation is responsible for administrative work group’s 


higher level of satisfaction.. 


E. SATISFACTION VERSES TIME OF SYSTEM USE 

Satisfaction with a computer system is expected to improve 
the longer users operate a system over time; gaining more 
experience and understanding of the system. 

An increase in overall satisfaction resulting from longer 
time of system use did not emerge from this study. The study 
did show, when considering the survey population as a whole, 
that user’s satisfaction remained constant when compared to 
length of time the respondent had used the system. 

As discussed in Chapter V, the Ancillary group making up 


46% of the survey population tends to influence all 
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generalizations drawn from total survey population findings. 
Breaking down the findings by work groups demonstrates this 
fact well. 

Positive trends in satisfaction for the administrative and 
physician work groups were displayed. Personnel in those work 
groups who have used the system the longest (greater than 11 
months) were more satisfied with CHCS than those who had used 
the system less than six months. 

The work performed by CHCS in the physician and 
administrative work groups was previously performed manually. 
As they worked with the system, their experience and 
confidence with CHCS expanded, thereby increased satisfaction 
was a result. 

Administrative personnel, satisfied with the training 
provided, quickly grasped the system procedures. The 
structured application of CHCS to their work performed 
assisted in their acceptance of the system. Rapid increases 
in satisfaction with CHCS resulted. 

The complex functionality of the physician modules is 
possibly responsible for their relatively prolonged time 
period before satisfaction is increased. After physicians 
learned the multitude of inputs necessary, their confidence 
and satisfaction with CHCS improved. 

Ancillary personnel findings indicate a steady decrease in 
Satisfaction with CHCS for those who have used the system the 


longest when compared to those persons having less than six 


a), 


months exposure to system use. This is the opposite to what 
was anticipated. It is not clear why this finding occurred. 
An in-depth analysis of their policies and procedure along 
with interviews with the personnel is necessary to determine 


the cause of this finding. 
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VII. CONCLUSIONS AND RECOMMENDATIONS 


A. MEASURING COMPUTER SYSTEM EFFECTIVENESS 

In an environment of scarce financial resources, 
Organizations need to measure the effectiveness of computer 
systems to justify expenditures. Of the several direct and 
in-direct methods available to measure computer’ system 
effectiveness, measurement of user information satisfaction is 
a suitable, available and effective surrogate measure. The 
previously developed and validated user information 
Satisfaction questionnaire of Baroudi and Orlikowski (1988) is 
easily administered and provides sufficient responses to 
determine a baseline reference point for overall satisfaction 
as well as indicating potential problem areas. The survey is 
not a definitive evaluation of CHCS, but combined with further 
investigation, can be a powerful tool in the analysis and 


interpretation into the cause of user dissatisfaction. 


B. SATISFACTION CHARACTERISTICS OF CHCS 

Overall satisfaction is a generalization of all the 
characteristics that effect the satisfaction of the user. 
Along with overall satisfaction, the study specifically looked 
at the four factors that make up satisfaction: local MID staff 
and services, contractor services, knowledge and involvement, 


and information product. 
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1. Overall Satisfaction 

The survey results indicate the users are slightly 
Satisfied with CHCS at Naval Hospital, Charleston as a whole. 
The overall satisfaction scores do not differ significantly 
for the various types of personnel operating the system. In 
other words, the different ways (modules) in which people use 
the system did not affect their overall perception of the 
system. 

2. Areas of Satisfaction 

All three work groups rated the local MID staff and 
services as quite satisfactory. This is an indication that 
the local MID personnel is interacting well with the system 
users. The users perceive the local MID is taking care of 
their needs. 

The output of the system, or information product, is 
considered by the users as slightly satisfactory overall. 
What they receive from the system in the form of output meets 
their expectations. 

Although the overall satisfaction scores did not 
differ significantly, the administrative work group displayed 
higher satisfaction in many areas. The slightly higher 
Satisfaction can be traced to their higher perception of 
Satisfaction with the training provided. This leads to higher 
satisfaction in participation and understanding of the system. 


The manner in which the administrative group personnel are 
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trained or assisted is producing a more positive effect on 
their satisfaction than the other work groups. 
3. Areas of Dissatisfaction 

Factors that result in user dissatisfaction or 
Significantly lower satisfaction when compared to other 
factors are potential areas for concern. Further 
investigation into areas of dissatisfaction can provide 
context, history and insight for possible corrective measures. 

The respondents were most dissastified with the 
contractor services in respect to the time for new system 
developement and changes to the system. Although the exact 
cause of this dissatisfaction cannot be determined, most 
likely, the user’s perception of the time required for these 
contractor activities is much less than the reality of the 
Situation. 

4. Satisfaction with Time of System Use 

The expectation is that users will become more 
comfortable and confident, and therefore more satisfied, with 
CHCS as they gain increased experince using the system. This 
expectation was shown true for the physician and the 
administrative groups. However, the ancillary work group’s 
satisfaction with the system was lower for experienced users 


than the less experienced users. 
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C. RECOMMENDATIONS 

This study showed that the short-form questionnaire 
developed by Baroudi and Orlikowski (1988) is an effective 
means to measure user information satisfaction. The survey 
instrument is an appropriate instrument to document user 
satisfaction with the CHCS at other military medical 
facilities as well as documenting changes in user satisfaction 
at Naval Hospital, Charleston at a later time. 

A survey of this type is best conducted by an individual 
at the site. It is beneficial for the researcher to have 
first hand experience as a user of CHCS. Additionally, direct 
contact with individuals for interview by the researcher is 
beneficial for adding context to history. 

Additional investigation should be undertaken in several 
areas to add context and history. A larger and clearer 
picture may explain the reasons why the work groups had 
significant differences in their reported satisfaction to the 
various factors. These include: 

1. The administrative group’s overall higher satisfaction 

over the other work groups 

2. The physicians’ lowered perception of satisfaction in 

almost all areas 

3. The reasons for the respondents having negative 

perceptions of satisfaction with contractor’s services 

4. The reduction of satisfaction for experienced 


ancillary users from less experienced ancillary users 
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The Baroudi and Orlikowski short-form questionnaire (1988) 
should be used to measure UIS at other CHCS sites for 
comparison to the results obtained from this study at Naval 
Hospital, Charleston. The survey should also be conducted at 
a later time at Naval Hospital, Charleston to monitor for any 


changes in UIS. 
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APPENDIX A 
Part A: General Information 


Hospital Department: (Check one) 
General Administration 
Nursing Administration 
Dietary 

Emergency 

Laboratory 

Medical Clinic 
Inpatient Nursing 
Pharmacy 

Radiology 

Other (Specify): 








Job Description: (Check one) 
Corpsman (0000) 
Technician 

Nurse 

Pharmacist 

Physician 

Physician Assistant 
Other (Specify): 








Highest Level of Education: (Check one) 
High School Graduate 

Some College 

Bachelor’s Degree 

Some Graduate Work 

Master’s Degree 

Doctoral Degree 

Medical Degree 

Other (Specify): 








» 


ge: 





Gender: Male Female 
Length of time (in months) you have used CHCS: 


Have you used other computer systems before ? Yes 





If your answer was Yes to question 7, was it a health care 


information system ? Yes No 
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Part B: The Questionnaire 


This section of the survey conveys your own personal feelings 


concerning the use of the Composite Health Care system at Naval 
Hospital, Charleston. Please do not attempt to analyze the 
questions. Remember, there are no right or wrong answers. 


Please follow these instructions: 


ie 


Check each scale in the position that describes your evaluation 
of the factor being described. 


Check each scale, do not omit any. 

Check only one position for each scale. 

Check in the space, not between spaces. Bio, NOT THIS 
a? ane Xx 


Work rapidly. Rely on your first impressions. 


ANSWERS BASED ON YOUR OWN FEELINGS 


Relationship with the Management Information Department (MID) staff 


dissonant : : : : : : : : harmonious 


bad : : : : : : : : good 


Processing of requests for changes to existing systems 


fast : : : : : 2 : : Slow 


TE EEE 


untimely : : : : : : : : timely 


et eee 


Degree of training provided to users 


complete : : : : : : : : incomplete 


low : : : : : : : : high 


User’s understanding of systems 


insufficient : : : : : : : : sufficient 


complete : : : : : : : : incomplete 


User’s feeling of participation 


positive : : : : : : ; >: negative 


insufficient : : : : : : : : sufficient 


cme mmm SS —— EELS 


10. 


i te Oe 


EZ 


a Nc 


Attitude of the Management Information Department staff 


cooperative ;: : : : : : : ; belligerent 


SSS eee 


negative : : : : : : : : positive 


ee ee 


high : : : : : : : : low 


Superior : : : : : : : : “Site rron 


Relevancy of output information (to intended function) 


useful : : : $ : : : : useless 


cc EES ee 


relevant : : : : : : : ¢ irrelevant 


Accuracy of output information 


inaccurate : 3: F 3” 3" ee 9: accumaee 
Lows eae 
Precision of output information 
low s 3 3.8 a eee 
definite ; : : : : : : : uncertain 


Communication with the Management Information Department staff 


dissonant : : : : : : : : harmonious 


destructive : : : : : : : : productive 


Time required for new systems development 


unreasonable : : : : : : : ; reasonable 


acceptable : : : ; : : : : unacceptable 


Completeness of the output information 
sufficient : : : : : : : : insufficient 


adequate : : : : : : : : inadequate 


pm ee eo 


Thanks again for your cooperation 
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APPENDIX B 


' 
Composite Health Care System 
User Satisfaction Survey 


This survey is part of a study of the Composite Health Care System 
(CHCS) effectiveness in military hospital inpatient, outpatient and 
administrative settings. The purpose of the study is to obtain information 
about your perceptions of how well CHCS functions in your area. 
: Just a few minutes are required to fill out the brief general 
information and the 13 question survey. Your responses will be treated in 
complete confidentiality. The data gathered through the survey will 
provide valuable insights into the system’s strengths and weaknesses, and 
assist in future development, training and use. 


{ 


The success of this survey depends on receiving as many completed 
surveys as possible from users in all hospital areas. Your participation 
makes an important contribution towards this end. There are no right of 
wrong answers, only your candid response to each question. Please do not 
omit any of the questions in either section of the survey. 


Please return your completed survey to LT Powell in the Pharmacy 
Department in the addressed envelope provided. Thank you for your 
cooperation. 


LT L.E. Hurd, MSC 
Naval Postgraduate School 
Monterey, Ca 


| 
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APPENDIX C-1 


COMBINED RESULTS FOR DOCTORS VS ADMINISTRATIVE VS ANCILLARY 


TOTAL SATISFACTION AVERAGE INDEX 
DOCTOR 0.620879 2 oO) 
ADMIN NR doh Si Se 14.96 
ANCILLARY 0.742475 oO 
INDIVIDUAL QUESTION RESPONSES 
QUESTION 1 Z 6 4 S 
DOCTOR 1.380952 -0.45238 0.047619 0.142857 )O352ee8 
ADMIN 1.897059 -0.08824 1.426471 1.411765 1° 58e2e.8 
ANCILARY 1.652174 -0.82609 0.184783 1.032609 0.913043 
STAND DEV. 
DOCTOR 1.28694 1.328898 1.342593 1.601393 1.6] 
ADMIN 1.198752 1.967916 1.586474 1.498609 12 30G08 
ANCILARY 1.287597 1.700735 1.803107 1.712598 1.67ileee 
QUESTION 6 7 8 9 Le 
DOCTOR 1.571429 1.071429 0.571429 1.404762 0. 857ieae 
ADMIN 2.308824 1.029412 1.220588 1.117647 07 805Gee 
ANCILARY 1.934783 0.48913 0.902174 1.097826 0.206528 
STAND DEV. 
DOCTOR 1.085219 0.921104 1.399353 1.083345 022 
ADMIN 0.965948 1.392631 1.401977 1.766432 1.650488 
ANCILARY 1.014232 1.72566 1.437793 1.576334" 25658 
QUESTION aval eZ LS 
DOCTOR 1.357143 -0-92857 032 
ADMIN 1.941176 -0.073535 077 James 
ANCILARY 1.576087 0.01087 0.478261 
STAND DEV. 
DOCTOR 1.185716 1.471566 @2ess38 
ADMIN 1.144518 1.764629 1 .55¢7 Ze 
ANCILARY 1.178877 1.775874 1.660652 
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PHYSICIANS vs THE REST OF THE SURVEY GROUP 


GENERAL 


DOCTOR SAT 
STDEV 
NON-DOC SAT 
STDEV 


GROUPED FACTORS 


DOCTOR GROUP 
NON-DOC GROUP 
DOC STDEV 
NON-DOC STDEV 


QUESTION 


DOCTOR SATIS 
NON-DOC SATIS 
DOC STDEV 
NON-DOC STDEV 


QUESTION 


DOCTOR SATIS 
NON-DOC SATIS 
DOC STDEV 
NON-DOC STDEV 


QUESTION 


DOCTOR SATIS 
NON-DOC SATIS 
DOC STDEV 
NON-DOC STDEV 


APPENDIX C-2 


AVERAGE INDEX 
0.620879 807 
1.445 Oreo 
Ors o0208 U2 610), 
i 03 12.44 


(MID) 


ee oe 


iL 


1580952 
eto Z) 
1.28694 

1.252654 


6 


1.571429 
2093 1D 
v0 S529 
DORON i bias. 


11 


IoD 7143 

Lo S25 
ie SomekG 
1 1A 7 98 


-44 
. 86 
eet i 
5 AL 1 


=0 
=O 


Z 


-0.45238 
=03 5125 
1.328695 
1.849656 


7 


1.071429 
One 75 
0.921104 
T-61039 


12 
=0', 92657 
—OR02) 


1.471566 
1.766032 


qa 


527 


(CONT) 
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82 
4l 


3 


0.047619 

OcgiZ5 
1 «342595 
Lrene7 2 


8 


0.571429 

pa ES i is, 
lL S2g853 
1.426997 


a 


02523161 
O@GiZ5 
078259591 
1.617466 


026685714 
Oe 6Gl773 
Paez 07 
1.604875 


4 


Omt42857 
Jk os Ey be 
ero 03.9.3 
1.631054 


9 


1.404762 
ito 625 
1.083345 
1.654409 


0.065476 


0) Gis: 


1.505041 
1.856174 


> 


OoZse 
1 20625 
1.611525 
Je es 5) s) 


EO) 


0.657145 

0.4625 
Q- 92562 
1.624837 
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PHYSICIANS vs THE REST OF THE STUDY GROUP 
CALCULATED T-VALUE 


QUESTION T-VALUE 


id ae OG 

2 O21 69 

S woe ley SiS, 

4 = Oo 
5 =1.°734 

6 ali OD ee 
7 313 

8 Sl 35> 

S, Oo 's 
10 1 eh, 
atl = de 
a2 eo oO 
3 =e 2 41 
FACTOR 

A -1.467 
MID 

A On 982 
CONTRACTOR 

B =—0.354 

Cc Ni S)S)aL 
OVERALL Woo) 


* SIGNIFICANT AT ALPHA = 0.05 


TZ 


ANCILLARY 


GENERAL 


ANCIL SATIS 


NON-ANCIL SATIS 


ANCIL STDEV 


NON-ANCIL STDEV 


GROUPED FACTORS 


ANCIL GROUP 
NON-ANCI GROUP 
ANCIL STDEV 
NON-ANCI STDEV 


QUESTION 


ANCILLARY 
NON-ANCILLARY 
ANC ST DEV 
NON-ANC ST DEV 


QUESTION 


ANCILLARY 
NON-ANCILLARY 
ANC ST DEV 
NON-ANC ST DEV 


QUESTION 


ANCILLARY 
NON-ANCILLARY 
ANC ST DEV 
NON-ANC ST DEV 


tA 


PRNF 


ee 
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vs THE REST OF THE SURVEY GROUP 


AVERAGE INDEX 
0.742475 2. OD 
O. 967852 i276 
1.722284 alice 4 
to 95243 tht is) 
A A B C 
(MID) (CONT) 
ed -0.41 0.634783 0.328804 
i o2 =07 52 O'.335614 0.697 f2a7 
ue es: 1.621374 1.874259 
12256 SS) Leogbera 1.725529 
1. 2 5 4 = 
/652274 -0.82609 0.184783 1032609802913 043 
Wad -O0'.22927 G2280. 927273 lgeltciteils 
wzonaod |. 700755.) 1 BVOSTOgeeee 12598 127691827 
~252067 1.753942 1.636454 1.65105» 1 52 M616 
6 7 8 e, 10 
w2od763 0 .4698s "0 302i ae .097826 06206522 
HOw Ll O45455) 0. 992i 1.221273 0.827273 
POdazeZ sl.t2506 1.437795 1.576634 1.56569 
nOvUaSse L.2g29ouz 1.430077 1.542422 1.413299 
iLal 12 ales 
wa7oocr “O.010ew. 0). 476261 
-7FBLBZ -0. 40909 0. OB0909 
~178877 1.775874 1.660652 
-189648 1.70426 1.325442 
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ANCILLARY vs THE REST OF THE STUDY GROUP 


CALCULATED T-VALUE 


QUESTION T-VALUE 


i =O ke 
2 = 1 63 
=) =i. ~ 
a Omao9 
2 =O oo 
6 =0 .423 
7! pelle 7 te) 
8 =O 7250 
9 023956 
IG -1.964)°* 
ala =C2 oz 
ihZ ee ook 
ae -0.674 
FACTOR 
A -0.406 
MID 
A =O: .245 
CONTRACTOR 
B -0.948 
Cc =O “976 
OVERALL Sa pg 7) 


* SIGNIFICANT AT ALPHA = 0.05 
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APPENDIX C-6 


ADMINISTRATION vs THE REST OF THE SURVEY GROUP 


GENERAL AVERAGE INDEX 
ADMIN SAT ene 2.27 14.96 
NON-ADMIN SAT 0.704363 12.94 
ADMIN STDEV 1.6444 Jeo 
NON-ADMIN STDEV Pee45581 O12 
GROUPED FACTORS A B C 

(MID) (CONT) 
ADMIN GROUP 2.05 Og O2994t1s 1.084550 
NON-ADMIN GROUP TOS sO es: 0.664463 0.244403 
ADMIN ST DEV ile es 1.86 1.552052 1.742720 
NON-ADMIN ST DEV ieee Al, Ssloye' 1.486632 1.770465 
QUESTION 1 2 S 4 3) 
ADMIN Poss —0.08824 1.426471 12411765 meee sZ235 
NON-ADMIN fear e4—-0 70896 0.141791 0. 7a3s73sl 0a 91045 
ADMIN STDEV Tale oeel. [677808 |. 5864 Am 498609 Te3060L 
NON-ADMIN STDEV1.265863 1.751332 1.746904 1.676404 1.605592 
QUESTION 6 7 8 9 1g) 
ADMIN 2.508824 1.029412 1.220588 "1.117647 0.808824 
NON-ADMIN 1.820896 0.671642 0.798507 1.19403 0.410448 
ADMIN STDEV Ge 965946 127392637 1.401977 1.766432 1.650485 
NON-ADMIN STDEV 1.04371 1.498369 1.430456 1.555407 1.512961 
QUESTION 11 12 13 
ADMIN 1.941176 -0.08824 0.794118 
NON-ADMIN ie 465 —0. 260558" 0,..4925357 
ADMIN STDEV 44528 1.764629 1.550723 
NON-ADMIN STDEV1.183936 1.74471 1.487448 
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ADMINISTRATION vs THE REST OF THE STUDY GROUP 


CALCULATED T-VALUE 


QUESTION T-VALUE 


a ie AS)”, 
2 1.614 
S) Seo 
4 12929 
5 2a50S = 
6 2.2 toy * 
7 1.162 
8 1.411 
9 -0.221 
10 1.23 
Bie net si, 
i On53 
i 0.949 
FACTOR 
A Logg 
MID 
A Toss 
CONTRACTOR 
B i O'S.6 
C 2.20504 
OVERALL 02955 


* SIGNIFICANT AT ALPHA = 0.05 
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APPENDIX C-8 


SATISFACTION BASED ON TIME ON THE SYSTEM 


OVERALL 
TIME OF 
# PEOPLE USE AVG SAT 
28 aro 5 11.64286 
36 6 TO 11 11.84722 
67. >= 12 10.37838 
10d: 
ADMINISTRATION 
TIME OF 
# PEOPLE USE AVG SAT 
8 1 TO 5 8.8125 
9 6 TO 11 17.4375 
naw >= 12 17.36111 
34 
ANCILLARY 
TIME OF 
# PEOPLE USE AVG SAT 
2 L.ro.5 16.91667 
19 6 TO 11 11.5 
15 =) 12 1.533333 
46 
DOCTORS 
TIME OF 
# PEOPLE USE AVG SAT 
8 mM TO. S 6.5625 
8 6 TO 11 6.375 
5 >=12 13.2 
21 
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